Add Db2/MSSQL/Oracle engines and TPC-H, Wikipedia, YCSB, CH-benCHmark benchmarks#21
Add Db2/MSSQL/Oracle engines and TPC-H, Wikipedia, YCSB, CH-benCHmark benchmarks#21ribalba wants to merge 3 commits into
Conversation
… benchmarks Restructures the repo into two trees: benchmarks/ holds the GMT usage scenarios (one folder per benchmark), db/ holds the per-engine driver scripts (one folder per engine). Replaces the three root usage_scenario_*.yml files. Engines: adds IBM Db2, Microsoft SQL Server and Oracle Database Free alongside the existing Postgres/MariaDB/MySQL. Db2 needs a one-time image build (db/db2/build-image.sh) because HammerDB's db2tcl binding needs the full Db2 client. Benchmarks: adds TPC-H (HammerDB TPROC-H) plus Wikipedia, YCSB and CH-benCHmark driven by a new BenchBase container. Every engine gets an equal 4-CPU / 8-GB budget in compose.yml so the energy comparison is fair. Tooling: - run_on_cluster.py discovers every scenario and submits it to the GMT cluster, refusing to run on an unclean or unpushed tree. - check_repo.py enforces the fairness invariants: the per-benchmark compose.yml copies match the root, DB resource budgets are equal, and the driver scripts use the same knobs across engines. All scenarios here are stock/default engine configuration; tuning follows in a separate PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
📝 WalkthroughWalkthroughThe repository now supports HammerDB and BenchBase workloads across PostgreSQL, MariaDB, MySQL, Oracle, Microsoft SQL Server, and Db2. It adds shared Compose environments, TPC-C, TPC-H, Wikipedia, YCSB, and CH-benCHmark scenarios, engine-specific driver scripts and XML configurations, Db2 and BenchBase image-build tooling, repository consistency checks, expanded setup documentation, and a cluster submission utility. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (7)
benchmarks/benchbase/Dockerfile (1)
74-85: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winRun the benchmark driver as a non-root user.
The runtime container executes scenario-supplied shell commands as root. Create a dedicated user, make
/benchbasewritable by it, and switch withUSER; verify any mounted result paths remain writable.Source: Linters/SAST tools
benchmarks/chbenchmark/compose.yml (2)
95-95: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPin
db2,hammerdb, andbenchbaseimages instead of:latest.Every DB engine here is pinned to an exact version (
postgres:18.4-trixie,mysql:9.7.1-oraclelinux9, etc.) excepticr.io/db2_community/db2:latest,tpcorg/hammerdb:latest, andribalba/benchbase:latest. For a benchmark suite whose stated goal is a fair, reproducible comparison, an unpinned driver/engine image can silently change behavior or performance between runs.Also applies to: 125-131
86-91: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winMSSQL healthcheck only confirms the TCP port is open, not that SQL Server is ready to serve queries.
mssql/serverimages are known to open port 1433 before the engine has finished first-run initialization; a bare TCP check can report "healthy" whilesqlcmd/schema-build steps would still fail. Consider a login-level check (e.g.sqlcmd -Q "SELECT 1") if the image shipssqlcmd.benchmarks/tpcc/db2.yml (1)
1-1: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDb2 image pinned to mutable
:latesttag. Both Db2 scenario files referenceribalba/hammerdb-db2:latest, a personal Docker Hub namespace with no immutable tag/digest. Since this PR's goal is fair, reproducible cross-engine comparisons, alatesttag that can change between the image build and the actual benchmark run undermines reproducibility.
benchmarks/tpcc/db2.yml#L19-23: pin the image to a specific tag or digest produced bydb/db2/build-image.sh.benchmarks/tpch/db2.yml#L24-28: same fix.db/mssql/tpcc/mssql_tprocc_buildschema.tcl (1)
16-17: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winCentralize benchmark database credentials. Every listed driver hard-codes a predictable login/password pair. Inject credentials from Compose or deployment-time environment/configuration, fail fast when absent, and reserve any default only for explicitly isolated local runs.
db/mssql/tpcc/mssql_tprocc_buildschema.tcl#L16-L17: replace the embedded MSSQL credential with the runtime value.db/mssql/tpcc/mssql_tprocc_deleteschema.tcl#L16-L17: use the same runtime credential for cleanup.db/mssql/tpcc/mssql_tprocc_run.tcl#L16-L17: use the runtime credential for workload execution.db/mssql/tpch/mssql_tproch_buildschema.tcl#L16-L17: replace the embedded MSSQL credential with the runtime value.db/mssql/tpch/mssql_tproch_deleteschema.tcl#L16-L17: use the same runtime credential for cleanup.db/mssql/tpch/mssql_tproch_run.tcl#L16-L17: use the runtime credential for workload execution.db/mysql/tpcc/mysql_tprocc_buildschema.tcl#L15-L16: replace the embedded MySQL credential with the runtime value.db/mysql/tpcc/mysql_tprocc_deleteschema.tcl#L11-L12: use the same runtime credential for cleanup.db/mysql/tpcc/mysql_tprocc_run.tcl#L11-L12: use the runtime credential for workload execution.run_on_cluster.py (2)
48-51: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winType
fail()asNoReturninstead of returning"None".
fail()always exits; typing itNoReturn(fromtyping) removes the need for the# type: ignore[union-attr]workaround at line 70 and lets type checkers correctly narrowspec/spec.loaderafter everyfail(...)call site in this file.Proposed fix
+from typing import NoReturn ... -def fail(message: str) -> "None": +def fail(message: str) -> NoReturn: """Print an error to stderr and exit non-zero.""" print(f"error: {message}", file=sys.stderr) sys.exit(1)
161-165: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
build_nameacceptsbranchbut never uses it, so scenario names collide across branches.Two different branches submitting the same scenario both get the name
"{prefix} tpcc/pg"; branch is preserved separately in the payload (line 351), but the display name itself can't distinguish runs by branch.def build_name(prefix: str, rel_path: Path, branch: str) -> str: - """A useful, unambiguous run name, e.g. 'DBMS-bench tpcc/pg'.""" + """A useful, unambiguous run name, e.g. 'DBMS-bench tpcc/pg (main)'.""" benchmark = rel_path.parent.name # tpcc / tpch (drop the benchmarks/ prefix) db = rel_path.stem # pg, maria, ... - return f"{prefix} {benchmark}/{db}" + return f"{prefix} {benchmark}/{db} ({branch})"
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 25ce505e-4633-49fc-88f4-f7b5b2f2fb50
📒 Files selected for processing (95)
README.mdbenchmarks/benchbase/Dockerfilebenchmarks/benchbase/build-image.shbenchmarks/chbenchmark/compose.ymlbenchmarks/chbenchmark/maria.ymlbenchmarks/chbenchmark/mssql.ymlbenchmarks/chbenchmark/mysql.ymlbenchmarks/chbenchmark/oracle.ymlbenchmarks/chbenchmark/pg.ymlbenchmarks/tpcc/compose.ymlbenchmarks/tpcc/db2.ymlbenchmarks/tpcc/maria.ymlbenchmarks/tpcc/mssql.ymlbenchmarks/tpcc/mysql.ymlbenchmarks/tpcc/oracle.ymlbenchmarks/tpcc/pg.ymlbenchmarks/tpch/compose.ymlbenchmarks/tpch/db2.ymlbenchmarks/tpch/maria.ymlbenchmarks/tpch/mssql.ymlbenchmarks/tpch/mysql.ymlbenchmarks/tpch/oracle.ymlbenchmarks/tpch/pg.ymlbenchmarks/wikipedia/compose.ymlbenchmarks/wikipedia/maria.ymlbenchmarks/wikipedia/mssql.ymlbenchmarks/wikipedia/mysql.ymlbenchmarks/wikipedia/oracle.ymlbenchmarks/wikipedia/pg.ymlbenchmarks/ycsb/compose.ymlbenchmarks/ycsb/maria.ymlbenchmarks/ycsb/mssql.ymlbenchmarks/ycsb/mysql.ymlbenchmarks/ycsb/oracle.ymlbenchmarks/ycsb/pg.ymlcheck_repo.pycompose.ymldb/db2/Dockerfile.hammerdb-db2db/db2/build-image.shdb/db2/tpcc/db2_tprocc_buildschema.tcldb/db2/tpcc/db2_tprocc_deleteschema.tcldb/db2/tpcc/db2_tprocc_run.tcldb/db2/tpch/db2_tproch_buildschema.tcldb/db2/tpch/db2_tproch_deleteschema.tcldb/db2/tpch/db2_tproch_run.tcldb/maria/chbenchmark/maria_chbenchmark_config.xmldb/maria/tpcc/maria_tprocc_buildschema.tcldb/maria/tpcc/maria_tprocc_deleteschema.tcldb/maria/tpcc/maria_tprocc_run.tcldb/maria/tpch/maria_tproch_buildschema.tcldb/maria/tpch/maria_tproch_deleteschema.tcldb/maria/tpch/maria_tproch_run.tcldb/maria/wikipedia/maria_wikipedia_config.xmldb/maria/ycsb/maria_ycsb_config.xmldb/mssql/chbenchmark/mssql_chbenchmark_config.xmldb/mssql/tpcc/mssql_tprocc_buildschema.tcldb/mssql/tpcc/mssql_tprocc_deleteschema.tcldb/mssql/tpcc/mssql_tprocc_run.tcldb/mssql/tpch/mssql_tproch_buildschema.tcldb/mssql/tpch/mssql_tproch_deleteschema.tcldb/mssql/tpch/mssql_tproch_run.tcldb/mssql/wikipedia/mssql_wikipedia_config.xmldb/mssql/ycsb/mssql_ycsb_config.xmldb/mysql/chbenchmark/mysql_chbenchmark_config.xmldb/mysql/tpcc/mysql_tprocc_buildschema.tcldb/mysql/tpcc/mysql_tprocc_deleteschema.tcldb/mysql/tpcc/mysql_tprocc_run.tcldb/mysql/tpch/mysql_tproch_buildschema.tcldb/mysql/tpch/mysql_tproch_deleteschema.tcldb/mysql/tpch/mysql_tproch_run.tcldb/mysql/wikipedia/mysql_wikipedia_config.xmldb/mysql/ycsb/mysql_ycsb_config.xmldb/oracle/chbenchmark/oracle_chbenchmark_config.xmldb/oracle/tpcc/oracle_tprocc_buildschema.tcldb/oracle/tpcc/oracle_tprocc_deleteschema.tcldb/oracle/tpcc/oracle_tprocc_run.tcldb/oracle/tpch/oracle_tproch_buildschema.tcldb/oracle/tpch/oracle_tproch_deleteschema.tcldb/oracle/tpch/oracle_tproch_run.tcldb/oracle/wikipedia/oracle_wikipedia_config.xmldb/oracle/ycsb/oracle_ycsb_config.xmldb/pg/chbenchmark/pg_chbenchmark_config.xmldb/pg/tpcc/pg_tprocc_buildschema.tcldb/pg/tpcc/pg_tprocc_deleteschema.tcldb/pg/tpcc/pg_tprocc_run.tcldb/pg/tpch/pg_tproch_buildschema.tcldb/pg/tpch/pg_tproch_deleteschema.tcldb/pg/tpch/pg_tproch_run.tcldb/pg/wikipedia/pg_wikipedia_config.xmldb/pg/ycsb/pg_ycsb_config.xmlpg/benchmark_tests.tclrun_on_cluster.pyusage_scenario_maria.ymlusage_scenario_mysql.ymlusage_scenario_pg.yml
💤 Files with no reviewable changes (4)
- usage_scenario_maria.yml
- usage_scenario_pg.yml
- usage_scenario_mysql.yml
- pg/benchmark_tests.tcl
| mariadb: | ||
| image: mariadb:13.0.1-rc | ||
| container_name: mariadb_container | ||
| cpus: 4 | ||
| mem_limit: 8g | ||
| environment: | ||
| MYSQL_ROOT_PASSWORD: maria | ||
| MYSQL_DATABASE: maria | ||
| MYSQL_USER: maria | ||
| MYSQL_PASSWORD: maria | ||
| # ports: | ||
| # - "3306:3306" | ||
| healthcheck: | ||
| test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] | ||
| interval: 10s | ||
| timeout: 5s | ||
| retries: 10 | ||
| start_period: 30s |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Is MariaDB 13.0.1 a stable GA release or a release candidate?
💡 Result:
MariaDB 13.0.1 is classified as a Release Candidate (RC) [1][2][3]. It is not a stable Generally Available (GA) release [1][4]. Official MariaDB documentation and release records explicitly label the 13.0.1 version, released on May 29, 2026, as an RC release [1][5][4].
Citations:
- 1: https://mariadb.com/docs/release-notes/community-server/13.0/13.0.1
- 2: https://mariadb.com/docs/release-notes/latest-releases
- 3: https://mariadb.com/docs/release-notes/community-server/changelogs/13.0
- 4: https://mariadb.org/mariadb/all-releases/
- 5: https://mariadb.com/docs/release-notes/community-server/changelogs/13.0/13.0.1
Use a GA MariaDB tag here — 13.0.1-rc is still a release candidate, so this benchmark can produce non-representative results versus the other stable engine images.
| @@ -0,0 +1,50 @@ | |||
| --- | |||
| name: BenchBase MySQL CH-benCHmark benchmark | |||
| author: Didi Hoffmann <didi@green-coding.berlin) | |||
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Correct the author-email delimiter in all BenchBase scenarios.
Each address starts with < but ends with ). Replace the closing delimiter with >.
benchmarks/chbenchmark/mysql.yml#L3-L3: change)to>.benchmarks/chbenchmark/oracle.yml#L3-L3: change)to>.benchmarks/chbenchmark/pg.yml#L3-L3: change)to>.benchmarks/wikipedia/maria.yml#L3-L3: change)to>.benchmarks/wikipedia/mssql.yml#L3-L3: change)to>.
📍 Affects 5 files
benchmarks/chbenchmark/mysql.yml#L3-L3(this comment)benchmarks/chbenchmark/oracle.yml#L3-L3benchmarks/chbenchmark/pg.yml#L3-L3benchmarks/wikipedia/maria.yml#L3-L3benchmarks/wikipedia/mssql.yml#L3-L3
| name: HammerDB Postgres benchmark | ||
| author: Didi Hoffmann <didi@green-coding.berlin) | ||
| description: Benchmarks postgresql with HammerDB using TPC-C |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Wrong benchmark name/description: says "Postgres" in the MySQL scenario file.
This file is benchmarks/tpcc/mysql.yml but the name/description are copy-pasted verbatim from pg.yml and still describe a Postgres benchmark. Since these fields presumably label results for the multi-engine fairness comparison this PR is building, this will mislabel MySQL benchmark runs.
🐛 Proposed fix
-name: HammerDB Postgres benchmark
+name: HammerDB MySQL benchmark
author: Didi Hoffmann <didi@green-coding.berlin)
-description: Benchmarks postgresql with HammerDB using TPC-C
+description: Benchmarks mysql with HammerDB using TPC-C📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| name: HammerDB Postgres benchmark | |
| author: Didi Hoffmann <didi@green-coding.berlin) | |
| description: Benchmarks postgresql with HammerDB using TPC-C | |
| name: HammerDB MySQL benchmark | |
| author: Didi Hoffmann <didi@green-coding.berlin) | |
| description: Benchmarks mysql with HammerDB using TPC-C |
| command: ./hammerdbcli auto /tmp/repo/db/mysql/tpch/mysql_tproch_buildschema.tcl | tee /dev/stderr | awk '/query [0-9]+ completed in/{q++} END{if(q>0){"date +%s"|getline ts; printf("%s000000 tpch_queries=%d\n", ts, q)}}' | ||
| shell: sh | ||
| log-stdout: true | ||
| read-sci-stdout: true | ||
| - name: Run Queries | ||
| container: hammerdb_container | ||
| commands: | ||
| - type: console | ||
| note: RUN HAMMERDB QUERIES | ||
| command: ./hammerdbcli auto /tmp/repo/db/mysql/tpch/mysql_tproch_run.tcl | tee /dev/stderr | awk 'END{"date +%s"|getline ts; printf("%s000000 tpch_queries=22\n", ts)}' | ||
| shell: sh | ||
| log-stdout: true | ||
| read-sci-stdout: true | ||
| - name: Drop Schema | ||
| container: hammerdb_container | ||
| commands: | ||
| - type: console | ||
| note: RUN HAMMERDB DROP | ||
| command: ./hammerdbcli auto /tmp/repo/db/mysql/tpch/mysql_tproch_deleteschema.tcl | tee /dev/stderr | awk '/query [0-9]+ completed in/{q++} END{if(q>0){"date +%s"|getline ts; printf("%s000000 tpch_queries=%d\n", ts, q)}}' |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Preserve HammerDB failures before publishing TPC-H work metrics.
Each tee | awk pipeline returns awk’s status rather than hammerdbcli’s. A failed driver run therefore succeeds at the scenario level; the query stage additionally emits tpch_queries=22 even when zero queries ran. Capture and check HammerDB’s exit status first, then derive the metric from validated output (or fail when the expected 22 completions are absent).
benchmarks/tpch/mysql.yml#L30-L48: make all three HammerDB stages propagate driver failures; only emit the run metric after validating completed queries.benchmarks/tpch/oracle.yml#L34-L52: make all three HammerDB stages propagate driver failures; only emit the run metric after validating completed queries.benchmarks/tpch/pg.yml#L30-L48: make all three HammerDB stages propagate driver failures; only emit the run metric after validating completed queries.
📍 Affects 3 files
benchmarks/tpch/mysql.yml#L30-L48(this comment)benchmarks/tpch/oracle.yml#L34-L52benchmarks/tpch/pg.yml#L30-L48
| commands: | ||
| - type: console | ||
| note: BENCHBASE EXECUTE | ||
| command: cd /benchbase/benchbase-mysql && java -jar benchbase.jar -b wikipedia -c /tmp/repo/db/mysql/wikipedia/mysql_wikipedia_config.xml --create=false --load=false --execute=true | tee /dev/stderr | awk 'match($0,/measuredRequests=[0-9]+/) {v=substr($0,RSTART+17,RLENGTH-17)} END {if (v!="") {"date +%s"|getline ts; printf("%s000000 wikipedia_requests=%s\n", ts, v)}}' |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not mask failed benchmark runs or missing functional-unit metrics.
Under sh, this pipeline returns awk’s status, so a failed java process can still mark the scenario successful. awk also exits zero when no measuredRequests line is found, producing a successful run without its SCI metric. POSIX defines the pipeline status as the rightmost command’s status when pipefail is not enabled. (pubs.opengroup.org)
benchmarks/wikipedia/mysql.yml#L47-L47: capture BenchBase output, fail on Java failure, and fail whenmeasuredRequestsis absent.benchmarks/wikipedia/oracle.yml#L47-L47: apply the same guarded execution.benchmarks/wikipedia/pg.yml#L47-L47: apply the same guarded execution.benchmarks/ycsb/maria.yml#L47-L47: apply the same guarded execution.benchmarks/ycsb/mssql.yml#L47-L47: apply the same guarded execution.benchmarks/ycsb/mysql.yml#L47-L47: apply the same guarded execution.benchmarks/ycsb/oracle.yml#L47-L47: apply the same guarded execution.benchmarks/ycsb/pg.yml#L47-L47: apply the same guarded execution.
Use a temporary output file: run java into it, print it to stderr, abort on a non-zero Java exit, then parse the file with awk whose END block exits non-zero when v is empty.
📍 Affects 8 files
benchmarks/wikipedia/mysql.yml#L47-L47(this comment)benchmarks/wikipedia/oracle.yml#L47-L47benchmarks/wikipedia/pg.yml#L47-L47benchmarks/ycsb/maria.yml#L47-L47benchmarks/ycsb/mssql.yml#L47-L47benchmarks/ycsb/mysql.yml#L47-L47benchmarks/ycsb/oracle.yml#L47-L47benchmarks/ycsb/pg.yml#L47-L47
| Notes: | ||
|
|
||
| - The Db2 server container requires `privileged: true`, and the host must allow it. | ||
| - Db2 is slow to start (minutes) and loads via INSERTs, so it builds slower than the bulk-loaded engines. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd -e yml . benchmarks/tpcc benchmarks/tpch --exec rg -n -A2 -B2 'privileged|cap_add' {} \;
rg -n -A2 -B2 'privileged|cap_add' compose.ymlRepository: green-coding-solutions/DBMS-bench
Length of output: 1401
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== README excerpt ==\n'
sed -n '60,78p' README.md
printf '\n== Db2 compose files ==\n'
for f in benchmarks/tpcc/db2.yml benchmarks/tpch/db2.yml compose.yml; do
echo "--- $f ---"
sed -n '1,180p' "$f"
doneRepository: green-coding-solutions/DBMS-bench
Length of output: 13513
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== locate Db2 service definitions ==\n'
rg -n -A4 -B4 'db2|privileged|cap_add|CAP_IPC_OWNER|sysctl' benchmarks/tpcc/db2.yml benchmarks/tpch/db2.yml compose.ymlRepository: green-coding-solutions/DBMS-bench
Length of output: 13362
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for p in [Path('benchmarks/tpcc/db2.yml'), Path('benchmarks/tpch/db2.yml'), Path('compose.yml')]:
print(f"\n### {p}")
text = p.read_text()
for key in ['privileged', 'cap_add', 'CAP_IPC_OWNER', '--privileged']:
if key in text:
print(f"contains {key}")
PYRepository: green-coding-solutions/DBMS-bench
Length of output: 311
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# show exact lines around any Db2 capability notes in YAMLs
for f in benchmarks/tpcc/db2.yml benchmarks/tpch/db2.yml compose.yml; do
echo "--- $f ---"
nl -ba "$f" | sed -n '85,120p'
doneRepository: green-coding-solutions/DBMS-bench
Length of output: 243
Replace privileged: true with the actual Db2 capability requirement README.md:69-72 — the runtime compose setup only adds CAP_IPC_OWNER via cap_add; it doesn’t require full privileged: true.
|
Runs started to see that evrything works out |
2 participants
Split of #20 — part 1 of 2. This is the default-configuration half: every engine runs stock, nothing tuned beyond what's needed to make it run. The tuning tiers follow in part 2 (stacked on this branch).
Structure
The repo moves to two trees:
benchmarks/for the GMT usage scenarios (one folder per benchmark) anddb/for the per-engine driver scripts (one folder per engine). The three rootusage_scenario_*.ymlfiles are replaced bybenchmarks/<benchmark>/<db>.yml.Engines
Adds IBM Db2, Microsoft SQL Server and Oracle Database Free alongside Postgres/MariaDB/MySQL. Db2 needs a one-time image build (
db/db2/build-image.sh) because HammerDB'sdb2tclbinding requires the full Db2 client, not the free CLI driver — see the Db2 setup section in the README.Benchmarks
Adds TPC-H (HammerDB TPROC-H) plus Wikipedia, YCSB and CH-benCHmark, the latter three driven by a new BenchBase container (
benchmarks/benchbase/build-image.sh). There is no Db2 BenchBase profile, so Db2 has TPC-C/TPC-H only.Fairness
Every DB service gets an equal 4-CPU / 8-GB budget in
compose.yml, and the load drivers are left unconstrained so they never bottleneck generation.Tooling
run_on_cluster.py— discovers every scenario and submits it to the GMT cluster, refusing to run on an unclean or unpushed tree (the cluster measures GitHub, not your working tree).check_repo.py— enforces the fairness invariants: the per-benchmarkcompose.ymlcopies match the root source of truth, DB resource budgets are equal, and driver scripts use the same knobs across engines.check_repo.pypasses on this branch with one expected warning:tpch.degree_of_parallelisn't set for maria/mysql/db2, which have no parallel-query knob in HammerDB.🤖 Generated with Claude Code
Summary
benchmarks/and per-engine scripts/configurations intodb/.run_on_cluster.pyfor validated cluster submissions.check_repo.pyto verify compose consistency, resource fairness, and benchmark parameters.